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The inclusion in she audio-visual programs, progress checks, 
and laminated panels of actual or closely related test items 
and the utilization of these materials in a concentrated manner 
just prior to an exit testing period are a gross violation of 
the rationale underlying the normative concept of grade level 
achievement. 

To see if there were some method of analyzing the effects 
of teaching test items or closely related teat items on the 
test results, consultant help was requested from the Educational 
Testing Service, Princeton, New Jersey. Dr. Henry Dyer and 
Mr. William Angoff came to Texarkana, to review the situation. 

“Ac 

They suggested the following statistical procedures: 

a. The first step in the procedure is to make some Judgments 
about the "sameness' 1 of item' that appeared in both the 
Instructional materials and in the exit tests. These we 
shall call the "exposed" items. 

b. The second step is vo verify the foregoing judgments by 
making a statistical analysis to determine how the items 
identified as "exposed" behave in comparison with items 
not so identified. If it is found that the exposed items 
had become easier, then she exit answer sheet for those 
students will be rescored using only non -exposed items 

to determine an increment attributable to growth equi- 
valent to the increment that would have been obtained 
from the full-length test had no items become exposed, 

* See Education Testing Service memorandum in Appendix 
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c. The third step la to establish the magnitude of the real 
gains earned by students on the non-exposed Items. 

The consultants agreed to preside any assistance needed 
In analyzing the test results. 

The first step In determlng the effects for teaching 
test items or closely related test Items was to see how 
many test Items or closely related test Items were being 
taught. Dr. Dyer suggested some rules for Judging whether 
Items Included in the Instructional program and items 
from the test are to be considered the same. These rules 
were adopted and used In analyzing the instructional 
materials. The rules are given below. 

Rules for Judging Whether Two Items Are To Be 
Co nsidered the Same 

Two items are to be considered the same if; 

1. Their wording Is identical in all respects. 

Example: A. Which of these Is a way to find the 

circumference In inches of a circle 
with a 6-inch diameter? 

(1) 3 X 3.14 (3) 3 X 3 X 3.14 

(2) 6 X 3.14 (4) 2 X 6 X 3.14 

B. Which of these is a way to find the 
circumference in inches of a circle 
with a 6-inch diameter? 

(1) 3 X 3.14 (2) 6 X 3.14 (3) 3 X 3 X 3.14 

(4) 2 X 6 X 3.14 

(Note change in arrange -j-ant of options.) 

2. The wording of the stem and the wording of the correct 
response are identical; the other responses have been 
changed , 
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A. 


Same as above . 




B. 


Which of these is 
of a circle with 


a way to find the circumference 
a 6-inch diameter? 




(1) 3.14 X 3 


(3) 3 X 2.17 



(2) 6 X 3.14 (4) 2 X 6 X 2.1416 




3. The correct response is Identical and the main sense of 
the stem has been retained despite a minor change in 
word ing. 

A. Same as above. 

B. The number of inches in the circumference of a 
circle with a diameter of 6 inches is: 

(1) 6 X 3.14 (3) 3 X 3 X 3.14 

(2) 3 X 3.14 (4) 6 X 6 X 3.14 

4. The main sense of the whole item has been retained 
despite the fact t it has been re-stated in the 
negative. 

A. Same as above, 

B. The number of inches in the cir :uraference of a 
circle with a diameter of 6 inches is not : 

(t) 6 X 3.1416 (3) 3 X 3 X 3.14 

(2) 6 X 22 (4) 2 X 3 X 22 

7 7 

5. The main sense of the stem has been retained despite 
a minor change in wording; the correct response is 
identical, but any incorrect option has been changed 
or omitted. 

A. Same as above. 

B. The number of inches in the circumference of a 
circle having a 6-inch diameter can be found by 
which one of these? 

(1) 3 X 22 (2) 6 X 3.14 (3) 3 X 3 X 3.14 

7 

6. The item has been changed from a multiple-choice to a 
true-false format by retaining the stem of the multiple- 
choice item and incorporating in the stem one of the options 
(correct or incorrect). 

A. Same as abo<’e. 

B. The numbe* - of inches in the circumference of a circle 
with a 6-inch d'famete" is 3 X 3 X 3.14: 



TRUE 






FALSE 






I 
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Tha consultants from Educational Tasting Service also pro- 
vided u formula' for calculating tho percentage of unexposed itema 
that were needed to have a minimum reliability coefficient of .80 

ifc 

for that group of Items. From this percentage figure, the evalua- 
tors calculated the minimum number of unexposed items that must be 
available to be able to develop a possible procedure to use such 
items as a means of determining achievement gains for the students. 
The minimum number of unexpoeed items was calculated for each form 
and for each grade level of all tests used in the dropout prevention 
program. The formula is as follows: 

FORMULA 

X Y 

C “ 6 (1 - t ) C • Percentage of unexposed items 

< - * 8 ) r 

8 ■ Reliability coefficient of 

shortened version of tast or 
sub-test (minimum acceptable 
is .80). 
r 

t = Reliability coefficient of 

full length test or sub-test. 

Tables 2 and 3 provide information showing the results of 
the analysis made of the instructional materials to determine the 
extent test items had been exposed to students in the teaching 
process. 



* This formula was derived from the Spearman-Brown Formula. 
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Form 4 

Arithmetic Problem Solving 
Form 1 
Form 2 
Form 3 
Form 4 
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Iowa Tests of Basic Skills 
Sub-tests : 


Vocabulary 
Form 1 
Form 2 
Form 3 
Form 4 

Reading Comprehension 

* Form 1 

"* Form 2 
Form 3 
Form 4 

Arithmetic Concepts 
Form 1 
Form 2 
Form 3 
Form 4 

Arithmetic Problem Solving 
Form 1 
Form 2 
Form 3 
Form 4 
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The Information from the preceding tables shows that 
approximately three-fifths of the sub-tests vontain Items 
that had been exposed to teaching to the extent they were 
not usable to measure achievement gain. 

On the Iowa Tests of Basic Skills there are four sub- 
tests on each of the forms for each grade level. Of the 
possible 32 sub-tests for all four forms of the Iowa Teats 
of Basic Skills, 18 or 56.25 percent of the sub- tests had 
test items or closely related test items exposed in the 
teaching process to the extent they are not usable to plot 
scores on the uncxpoced items. Tables 4 and 5 provide an 
information summary concerning the percentage of test items 
on the Iowa Tests of Basic Skills that had been exposed In 
the teaching process. 
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Table 4. PERCENTAGE OF TEST ITEMS IN IOWA TESTS OF 3ASIC SKILLS, GRADE 7, 
THAT HAD BEEN EXPOSED IN THE TEACHING PROCESS 





VOCABULARY 






READING COMPREHENSION 






No. Items 


No, Test 






No. Items 


No. Test Items 


Form 


on Test 


Items Exposed 


7. 


Form 


on Test 


Exposed 


% 


1 


48 


48 


100.00 


1 (Pre 1 
Test) 


78 


0 


00.00 


2 


48 


48 


100.00 


2 


78 


33 


42.31 


3 


43 


48 


100.00 


3 


70 


35 


44.87 


4 


48 


48 


100.00 


4 


78 


50 


64.10 


TOTAL: 
ae 


192 


192 


100.00 


TOTAL: 


312 


118 


37.82 




ARITHMETIC 


CONCEPTS 




ARITHMETIC 


PROBLEM SOLVING 






No. Items 


No, Test Items % 




No. Items 


No# Test Item9 


Form 


on Test 


Exposed 




Form 


on Test 


Exposed 


7. 


1 


40 


17 


35-4! 


1 


32 


19 


59.37 


2 


43 


22 


45.83 


2 


32 


17 


53.12 


3 


48 


23 


47.91 


3 


32 


19 


59.37 


4 


48 


22 


45.83 


4 


32 


32 


100.00 


TOTAL: 


192 


84 


43. 75 


TOTAL: 


128 


87 


67,97 
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liable 5. PERCENTAGE OF TEST ITEMS in IONA TESTS OF BASIC SKILLS, GFADES 8-9- 
THAT HAD BEEN EXPOSED IN THE TEACHING PROCESS 





VOCABULARY 








READING COMPREHENSION 




No, Items 


No, Test 


Items 




No, Items 


No. Test 


Items 


form 


on Test 


Exposed 


7. 


Form 


on Test 


Exposed 


7 . 


1 


48 


48 


100.00 


l (Pre 


- 30 


0 


00.00 










Test) 








2 


48 


48 


100.00 


2 


00 


33 


41.75 


3 


48 


48 


100,00 


3 


GO 


35 


43.75 


4 


48 


48 


100,00 


4 


80 


38 


47.50 


TOTAL 


192 


192 


100.00 




320 


106 


33.12 


ARITHMETIC CONCEPTS 




ARITHMETIC 


PROBLEM SOLVING 




No, Items 


No, Test 


Items 




No. Items 


No. Test 


Items 


tom 


on Test 


Expos rJ 


7. 


Form 


on Test 


Exposed 


% 


1 


43 


16 


33,33 


1 


34 


21 


61.76 


2 


48 


22 


45.83 


2 


34 


24 


70.58 


3 


48 


22 


45.83 


3 


34 


25 


73.52 


4 


48 


23 


47.92 


4 


34 


34 


100.00 


TOTAL 


192 


83 


43.23 




136 


104 


76.47 
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: 



On the Science Research Associates Achievement Tests, 
there are five sub-tests on each of the forms for each grade 
level. Of the possible 20 sub-tests for the two forms of the 
Science Research Associates Achievement Tests, 12, or 60 per- 
cent of the sub-tests had test Items or closely related test 
Items exposed in the teaching process to the extent they are 
not usable to plot scores on the unexposed Items. Tables 6 
and 7 provide an Information summary concerning the percen- 
tage of test items on the Science Research Associates Achieve- 
ment Tests that had been exposed In the teaching process. 
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Table 6. PERCENTAGE OF TEST ITEMS in SCIENCE RESEARCH ASSOCIATES ACHIEVEMENT TESTS, 
GRADE 7, THAT HAD BEEN EXPOSED IN THE TEACHING PROCESS 





VOCABULARY 






READING COMPREHENSION 






No . Items 


No, Test Items 






No. Items 


No. Test 


Items 


Foma 


on Test 


Exposed 


7. 


Fora 


on Test 


Exposed 


7. 


C 


46 


38 


82.60 


C 


46 


30 


65.22 


D 


46 


30 


65.22 


D 


46 


30 


65.22 


TOTAL 


92 


68 


73.91 


TOTAL: 


92 


60 


65.22 




ARITHMETIC REASONING 






ARITHMETIC 


CONCEPTS 






No • Items 


No. Test Items 




No. Items 


No. Test 


Items 


Fora 


on Test 


Exposed 


7. 


Form 


on Test 


Exposed 


7. 


C 


43 


32 


74.42 


C 


47 


9 


19.15 


D 


43 


28 


65.12 


D 


47 


11 


23.40 


TOTAL: 


06 


60 


69.77 


TOTAL: 


94 


20 


21.28 




ARITHMETIC COMPUTATIONS 














No. Items 


No. Te9t Items 










Fora 


on Test 


Exposed 


X 










C 


47 


18 


38.30 










D 


47 


18 


38.30 










TOTAL: 


94 


36 


38.30 









o 



o 

ERIC 
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Table 7. PERCENTAGE OP TEST ITEMS In SCIENCE RESEARCH ASSOCIATES ACHIEVEMENT TESTS, 
GRADES 8 AND 9, THAT HAD BEEN EXPOSED IN THE TEACHING PROCESS 
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4. Summary Statement 

The achievement gain made by students as related to the 
number of hours In the instructional program was to be the 
basis of payment to the contractor. Achievement gain in 
reading was intended to be the average of the grade level 
increases earned by students on the vocabulary and reading 
comprehension tests. The achievement 3 aln in arithmetic was 
intended to be calculated by using the total of all the 
arithmetic sub-tests. 

Information presented in Tables 4, 5, 6, and 7 indicates 
that (a) relatively few sub-tests contained a sufficient number 
of unexposed items to provide a measuring instrument with a 
reliability coefficient of .80, (b) the sub-te9ts that seem 
to possess a sufficient number of unexposed item9 are not 
the same sub-tests on both' the Iowa Tests of Basic Skills 
and the Science Research Associates Achievement Tests, and 
(c) the sub-tests intended to measure reading and arithmetic 
gains are not now usable because one cr both have been invali- 
dated because of exposure of test items in the teaching process. 

Therefore, the teaching >f test items or closely related 
test items has invalidated the test results to the extent they 
cannot be used as a valid measure of achievement. 
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IV. PRODUCT EV/LU/T10M RESULTS 

In presenting the evaluation results, a general format will be 
followed. The objectives being evaluated will be listed, followed by 
the evaluation information. Where evaluation evidence was not available 
or was not valid, this will be stated. 

Objective One ; 

Ihe student in the rapid learning center program will display an 
increased vocabulary, reading comprehension, and arithmetic knowledge 
as indicated by scores on the Iowa Tests of Basic bkills or the Science 
Research Associates Achievement Tests. 

Results . Due to the contractor teaching for the test, no valid 
test results were available. 

Objective Two ; 

The students in the rapid learning center progrem will display an 
Increased knowldege of study skills as indicated by scores on the Science 
Research Associates Study Habits Checklist. 

Results . The pretest war administered to students in the rapid 
learning center program during the week of December 8-12, 1969. The 
post test was given to rapid learning center students during the first 
two weeks of May, 1970. To ensure uniformity in administering the 
Study Habits Checklist and to minimise the effects of deficient reading 
skills, the test administrator read the test questions aloud to all 
students , Ore hundred twenty-six students in the rapid learning center 
program took both the pre and post tests. The results are presented in 
Table 8. 
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Table 8. A COMPARISON OF THE FRE AND POST TEST SCORES OF 126 RAPID 
LEARNING CENTER STUDENTS ON THE SRA STUDY HABITS CHECKLIST 





Pretest 


Post test 


Difference 


t 


Mean 


86.5.'> 


87.81 


1.26 


.A3* 


S. D. 


23.35 


22.56 







* Not significant 



The data in Table C indicated that the rapid learning center 
students did not mike a statistically significant gain on the Study 
Habits Checklist. Therefore, objective two was not achieved. 

Objective Thre e; 

The students in the rapid learning center program will demonstrate 
increased application of pronunciation skills as indicated by scores on 
the Photo Articulation Test. 

Results . The pretest was administered to students in the rapid 
learning center program during the week of December 8-12, 1969. Tho 
post test was given to rapid learning center students during the week 
of May 11-15, 1570. The post tests were given only to those students 
who had articulation problems as indicated by their pretest scores. To 
ensure uniformity in administering the Photo Articulation Test, they were 
administered and scored by certified speech therapists. The tost was 
8lven on an individual basis to each student. Thirty-seven students in 
the rapid learning center program were glv^n both the pre and post 
Photo Articulation Tests. The results are contained in Table 9. 
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Table 9. A COMPARISON OF THE PRETEST AND POST TEST SCORES 
OF 37 STUDENTS ON THE FHOTO ARTICULATION TEST 





Number of 
Defective Sounds 
on Total Pretests 


Number of 
Defective Sounds 
on Total Post Tests 


Difference 


t 


Mean 


3.46 


3.89 


.43 


.65* 


3. D. 


2.27 


3.25 







* Not significant 



The data In Table 9 Indicates that the rapid learning center 
students did not show a statistically significant Improvement on the 
Photo Articulation Test. Objective three was not achieved. 

Objective Four : 

The students will display knowledge of the world of work by choosing 
at least one employment goal that is realistic and achievable as judged 
by the student's counselor. 

Results . When students took the post achievement test, the rapid 
learning center students were given a questionnaire asking them to 
Identify what Jobs they would like to get upon high school graduation. 
They were asked to rank the jobs in terms of their first, second, and 
third choice. This information was given to the students' counselors 
to make a Judgment on the feasibility of the choices made by the students, 
considering the counselors' knowledge of the students' interests, 
aptitudes, and abilities. 

The presrures of time prevented the counselors from providing the 
ratings of the students' vocational choices to the evaluator. Table 10 
presents the only available data concerning the occupational goals 
selected by students from the rapid learning center program and the 
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counselor's judgment of these goals. The sample is so small it is of 
questionable value. 



Table 10. THE COUNSELORS' RATINGS OP OCCUPATIONAL GOALS SELECTED 
BY STUDENTS IN THE RAPID LEARNING CENTER PROGRAM 





Choice 


I 


Choice 


II 


Choice 


III 


Ratings of Goals 
By Counselors 


Number 

Selecting 


Per- 

cent 


Number 

Selecting 


Per- 

cent 


Number Per- 
Selectlng cent 


Entirely unrealistic 


5 


13.2 


4 


11.4 


0 


.0 


Possible but unlikely 


to attain 


5 


13.2 


8 


22.8 


6 


19.3 


50-50 chance of 


attaining 


1 


2.6 


1 


2.9 


3 


9.7 


Might attain with 


extra effort 


5 


13.2 


2 


5.0 


2 


6.5 


Very realistic goal 


22 


57.8 


20 


57.1 


20 


66.5 


Total 


30 


100.0 


35 


100.0 


31 


100.00 



Objective F ive; 

The students will demonstrate an increased application of good 
grooming as Indicated by an appearance checklist. 

Results . The pretest is a grooming checklist developed by the 
Region VIII Education Service Center, Magnolia, Arkansas. The homeroom 
teacher of each student in the rapid learning center program was asked 
to rate him on the grooming checklist. Due to pressures of instructional 
duties and lack of time, the teachers were unable to respond to the 
request for ratings. Therefore, no data are available to evaluate this 
objective. 




75 



July 20, 1970 



Ob jective Six : 

The students will respond positively to the rapid learning center 
program as Indicated by: (1) a feedback questionnaire, (2) decrease In 

school absenteeism, (3) decrease In dropout frequency, and (4) Improving 
grades In other classes. 

Types of Information : A feedback questionnaire to be given students 

was developed by the Region VIII Education Service Center, Magnolia, 
Arkansas. School records were used to determine frequency of absences, 
dropouts, ar.d grade records. 

Data Collection Procedures : Students In the rapid learning center 

program were given a feedback questionnaire to determine their response 
to their school experiences. This questionnaire was given at the 
beginning of the program and again at the end of the program. Because 
of the nature of the program where students could be entered and exited 
at different times during the school year, only 67 students took both 
the ore and post questionnaire. 

The school records of the students In both the rapid learning 
center program and an equated group of non-rapid learning center stu- 
dents were analyzed to determine the frequency of absenteeism, the 
number who dropped out of school, and the grades earnsd during the 
school year. 

Results : The results of the pro and post questionnaires ere found 

In Table 11. The "always" is the most positive response for Questions 
Nos. 2, A, 7, 10, 11, 13, 17, 18, and 19, The "never" Is the most 
positive response for Questions Nos. 1, 3, 5, 6, 8, 9, 12, 14, 15, 
tnd 16. 

The scores on the feedback ouestlonnalre were quantified by 
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establishing a scoring system where 4 points were given for the ox> 3 t 
positive response, 3 points for the next, 2 for the next, and 1 point 
for the least positive response. Table 11 presents Information comparing 
the scores of the students on the pro test with their scoros on the post 
test. It will be noted that the information in Table 11 shows a slight 
numerical Increase in the positive attitude of the students towards the 
rapid learning center program. This difference Is not statistically 
significant. Further analysis Is needed regarding students' opinions, 
to determine what factors were significant In influencing their responses. 
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♦Students were asked to respond to the questionnaire in teras of the tepid Learning Center Program. 
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Table 12. A COMPARISON OF THE CHANGE IN RESPONSES ON A QUESTIONNAIRE TOWARD 

SCHOOL ACTIVITIES OF 67 STUDENTS IN THE RAPID LEARNING CENTER PROGRAM 



Mean 

S.D. 

* Not significant 



l* retest Responses 
55.71 
3.79 



Post Test Responses 
57.06 
8.15 



Differences "t," 
2.15 1.46* 



To explore the effects of the rapid learning center pro- 
gram on the school absenteeism of the students, the school 
attendance records ifere reviewed. Table 13 contains information 
concerning the mean number of days that the rapid learning center 
students could have attended the program, and the mean number 
of days that they actually attended. The difference shows that 
the rapid learning center students were absent from school an 
average of 10.47 days. 



Table 13. A COMPARISON OF NUMBER OF POSSIBLE DAYS OF SCHOOL ATTENDANCE WITH 
ACTUAL NUIGER OF DAYS OF ATTENDANCE FOR 351 RLC STUDENTS 





Mean 


Possible Days of Attendance 


79.46 


Actual Days of Attendance 


68.99 


Difference 


10.47 
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Because the dropout prevention program allowed students to be 
entered and exited at different Intervals, it was difficult to obtain 
absentee data on a comparable non-rapid learning center group of students. 

Data were available concerning the number of days absent for scu- 
dents in the rapid learning center and an equated group of non-rapid 
learning center students who took the February 2, exit test. Table 14 
provides information about the mean number of absences for each group 
of students. While the students in the rapid learning center have a 
slightly higher absentee rate, it is not significantly higher. The data 
on the letter of the tables suggest that the rapid learning center pro- 
gram has not had enough impact to decrease significantly the frequency 
of school absences. 



Table 14. A COMPARISON OF THE FREQUENCY OF ABSENTEEISM FOR STUDENTS 
IN THE RAPID LEARNING CENTER AND AN EQUATED NON-RAPID 
LEARNING CENTER GROUP, FEBRUARY 2 EXIT TESTING GROUP 



Group 


Number 


Mean Number of Absences 


Rapid Learning Center Students 


33 


6.68 


Equated Non-Rap’ d Learning Center 
Group of Students 


33 


5.53 


Difference 


MM 


1.15 


Mj.fl 


m m 


.833 



The overall goal of the program is the prevention of dropouts. To 
determine the effectiveness of the rapid learning center program in the 
prevention of dropouts, an analysis was made of the number of students 




80 



July 20, 1970 



dropping out of school from the rapid learning center program and the 
students from an equated non-rapid learning center group. This infor- 
mation is presented in Table 15. It can be seen from this information 
that nearly three times aa many students dropped out of school from the 
non-rapid learning center group than dropped out of school from the rapid 
learning center group. Whether this is a permanent trend can only be 
determined through a follow-up study. 



Table 15. A COMPARISON OP THE NUMBER 0? 8CKCCT, DROPOUTS FOR STUDENTS 

IN THE RAPID LEARNING CENTER PROGRAM AND STUDENTS IN THE EQUATED 
NON -RAPID LEARNING CENTER GROUP 



School 


Rapid Learning Center Group 


Equated Non-Rapid Learning 


Total No. 
in 

Group 


No. 

of 

Dropouts 


Per- 

Cent 

% 


Center Group 
Total No. No. 

in of 

Group. Dropouts 


Per- 

Cent 

% 


Arkansas Senior High 


54 


5 


9.25 


29 


5 


17.24 


College Hill Junior High 


75 


7 


9.33 


32 


5 


15.62 


Jefferson Junior High 


78 


6 


7.69 


45 


6 


13.33 


Liberty Eylau Senior High 


55 


1 


1.81 


59 


15 


25.42 


liberty Eylau Junior High 


41 


0 


.00 


0 


0 


.00 


Washington Junior High 


48 


5 


10.41 


70 


11 


15.71 


TOTAL 


351 


24 


6.84 


235 


42 


17.37 



To further analyze the extent to which the rapid learning center 
program was preventing dropouts, a comparison was made between the fre- 
quency of dropouts from the rapid learning center group and the frequency 
of school dropouts from the entire student population of the participating 
school districts. This information is found in Table 16, on the following 
page . 
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Table 16. A COMPARISON OF THE NUMBER OF SCHOOL DROPOUTS OF STUDENTS 
IN THE RAPID LEARNING CENTER PROGRAM WITH ALL STUDENTS 
IN THE PARTICIPATING SCHOOL DISTRICTS (1969-70 SCHOOL YEAR) 



J'lly 20, 1970 
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It has been previously pointed out that a number of students were 
enrolled in the rapid learning center program who did not fulfill the 
entry criteria. An analysis was made of the frequency of the school 
dropouts of students from the rapid learning center group according to 
whether they fulfilled or did not fulfill the entry criteria. The entry 
criteria were too grades or more behind in mathematics and reading 
achievement, as indicated on standardized test scores, and having an 
IQ of 75 or higher on an intelligence test. Where test scores were 
not available on the students, they are classified in the non-criteria 
group. The data are presented in Table 17 on the following page. 



) 
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Table 17. AN ANALYSIS OF THE NUMBER OF SCHOOL DROPOUTS 
FROM THE RAPID LEARNING CENTER GROUP 
A.CCORDING TO WHETHER THEY FULFILLED ENTRY REQUIREMENT 



July 20, 
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The striking Information in the previous table is that there were, 
very few dropouts from the rapid learning center students who fulfilled 
entry criteria. Less than one percent of uhe students fell in this 
group. Information from the previous three tables indicates that the 
dropout prevention program did reduce the frequency of dropouts and was 
particularly effective for those students fulfilling the entry criteria. 

Because the United States Office of Education requested that the 
evaluation data be submitted by August 1, 1970, and because of the extra 
work involved in investigating the effects of the cont. ,ctor teaching 
the test, it was not possible to analyze the complete effects of the 
rapid learning center program on students 1 grades in other school subjects. 

However, first semester grades were available on those students who 
took the post tests on February 2, 1970. To see if the rapid learning 
center program had any effect on the students' grades in other school 
subjects to this time, a comparison was made of students' last year's 
spring semester grades with this year's fall semester grades. This was 
done for the students in the rapid learning center program and for the 
students in the equated non-rapid learning center group. This infor- 
mation is found in Table 13, on the following page. 




35 



Table 13. AN ANALYSIS OF GRADES EARNED 3Y STUDENTS IN THE RAPID LEARNING CENTER PROGRAM 
AND THE EQUATED NON-RAPID LEARNING CENTER GROUP 



July 20 , V 970 
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The number of students receiving grades for each subject area is 
not the same between groups or from semester to semester within a group. 
The reason for this is that some students did not take one of the four 
subjects each semester, and some of the students in the rapid learning 
center program were in that special program in place of their regular 
English, social studies, mathematics, or natural science classes. 

In looking at the information on the previous table, the following 
can be noted: (1) the English grades of the rapid learning center stu- 
dents appear to be improving while the English grades of the non-rapid 
learning center group are becoming lower; (2) grades in courses in social 
studies, natural science, and mathematics are continuing the trend of 
becoming poorer each semester for both groups* 

Objective Seven : 

The project director will display knowledge of the feasibility of 
a rapid learning center program for all students in the school system 
as indicated by: (1) feasible cost, (2) available space, (3) appro- 

priateness of rapid learning center material, (4) acceptance by faculty 
and parents, and (5) permanency of student achievement gains. 

Results on Cost Feasibility : The feasibility of the cost for 

operating a rapid learning center program was to be determined through 
a cost analysis study. This study was to compare the instructional 
costs for producing grade level increases of the rapid learning center 
students with the comparable costs for producing grade level increases 
for all the students in the participating school districts. Since the 
test results have been invalidated this study cannot be done. Early in 
July, 1970, Dorsett submitted to the project director a summary of costs 
for the dropout prevention program. The president of the company pointed 
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out, In an accompanying letter, that a number of costs were omitted 
and that data were submitted hurriedly so as to provide some cost infor- 
mation. The following presents the available cost data for operating 
the rapid learning center program: 

DORSETT EDUCATIONAL SYSTEMS, INCORPORATED 
TEXARKANA PROJECT 



Direct Instructional Salaries $ 49,820 

Resident Administrative Salaries 33,755 

Staff and Consulting while at Texarkana 9,612 

Direct Support Labor not at Texarkana 12,651 



TOTAL DIRECT LABOR. $105, 838 

(81% Net Payroll - 19% Burder) 

Direct Expenses, Materials, Travel, Per Diem 4,217 

TOTAL DIRECT COSTS $110,055 



Allocable General and Administrative, Indirect and 



and Overhead Costs 36.302 

TOTAL DIRECT AND ALLOCABLE COSTS $146,357 



Indirect Support Labor $ 26,718 

Indirect Support Expenses 6,411 

Indirect Support Overhead 10,812 



TOTAL INDIRECT SUPPORT COSTS $ 43,941 
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From the above cost data, an estimate can be made of the coat per 
student and the cost of operating the program per student hourly rate. 
This Information Is found In Table 19, on the following page. It should 
be pointed out that many expenses occur in starting a new project that 
would not recur In a continuing program. Thus, the cost figures In no 
way reflect the amount of expenses that would be required In a continuing 
project. They are presented to provide some limited concept of start-up 
costa for a new project. 
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Table 19. AN ANALYSIS OF THE COST OF OPERATING 
THE RAPID LEARNING CENTER PROGRAM AS 
DETERMINED ON PER STUDENT AND HOURLY BASIS 



1970 



July 20, 
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The space study was not conducted as the school districts are 
Involved in the process of totsl desegregation* Because of this 
cultural change some school buildings are being phased out of operation. 
The available space for operation of learning centers in Phase II will 
primarily be restricted to the mobile units and existing refurbished 
classrooms. 

The cur iculum materials used in the rapid learning center program 
was to be rated by a cadre of teachers from the participating school 
districts. The internal ^valuators developed a form to be used for 
the ratings. The project director was unable to secure this information 
and thus, it is not available for analysis. 

Results on Acceptance by Faculty and Parents : A feedback question- 

naire was given to all teachers In the participating school districts. 

The results of the questionnaire are found in Tables 20a and 20b. An 
analysis of the data in these tables suggests that the teachers are 
familiar with the rapid learning center program and are willing to try 
it out in the regular classes. Teachers who had students in their classes 
who were participating in the rapid learning center program felt that 
such stvidenta 1 attitudes toward school had improved, and they partici- 
pated more readily in class discussion, but had not improved their grades, 
study habits, or class attendance. Prom the viewpoint of the teachers, 
it appears that the climate for a change is good toward the dropout 
prevention program, including the use of teacher aides. 
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Table 20a. THE RESPONSES OP TEACHERS TO A FEEDBACK QUESTIONNAIRE 







Number Responding 


Question 


Yes 


No 


1. 


I have heard about the Rapid Learning Center Program 


220 


0 


2. 


I an familiar with the techniques used in the Rapid 
Learning Center. 


157 


52 


3. 


I have visited the Rapid Learning Center. 


154 


74 


4. 


I have students in my classes who are attending the 
Rapid Learning Center Program. (If yes, please 
respond to statements a through f.) 


73 


108 




a. Since attending the RLC program, the attitude 

of these students toward school has generally 
improved. 


44 


13 




b. Since attending the RLC, these students show 
greater concern for their personal appearance. 


15 


35 




c. These students participate more frequently in 
class discussion than they did before attending 
the RLC. 


30 


21 




d. These students have improved their study habits 
and use their time more efficiently. 


20 


25 




e. The students in the RLC have improved their 
grades in my class. 


16 


30 




f. Class absences of these students have 
decreased . 


26 


"7 
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Table 20b. THE EXTENT TEACHERS AGREE WITH STATEMENTS 
CONCERNING THE DROPOUT PREVENTION PROGRAM 



- Number R5 jLDpMlnR_ 

Statement SA A D SD 



1. 


Techniques used In the Rapid Learning Center 
will work In the regular classroom. 


27 


105 


28 


5 


2. 


The teaching machines and programmed lnstruc- 
tlonal materials used In the RLC can be 
adapted for use In the regular classroom 


41 


114 


13 


0 


3. 


The Rapid Learning Center program would be 
feasible for some classes but not all classes. 


28 


121 


19 


2 


4. 


I believe that with the help of an aide my 
students would achieve as much as In the RLC 
program. 


22 


66 


41 


16 


5. 


An extra attempt should be made to assist 
the underachiever. 


131 


59 


3 


0 


6. 


The addition of RLC equipment and materials 
to the regular classroom would enable me to 
more effectively teach the average and above 
average student as well ao the underachiever. 


51 


87 


20 


3 


7. 


The Rapid Learning Center program can be 
effective in preventing dropouts. 


42 


124 


4 


6 


8. 


The teacher workload would Increase with the 
addition of teaching machines, record keeping 
systems, etc. In the regular classroom. 


38 


96 


31 


0 


9. 


If given an aide to assist with extra clerical 
tasks, I would be willing to try RLC techniques 
in the classroom. 


59 


96 


8 


4 


10. 


The Rapid Learning Center Is a good Idea, but 
it should be kept separate from the regular 
classroom. 


17 


40 


82 


18 


11. 


I would like to learn more about the Rapid 
Learning Center program. 


91 


94 


0 


0 



KEY! SA 
A 
D 
SD 



Strongly Agree 

Agree 

Disagree 

Strongly Disagree 
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Parents of students participating In the rapid learning center 
program were requested to respond to a questionnaire. One hundred 
and eighteen parents responded. The questionnaire results are found 
In the following Table 21, 

Table 21. AN ANALYSIS OF PARENT RESPONSES ON A QUESTIONNAIRE CONCERNING 
THE RAPID LEARNING CENTER PROGRAM 



ITEM 



Percentage of 
Responses 




Have you heard about this special school program your 

child has attended? Yes 97.417, N o 2,597 . 

Have you visited the Rapid Learning Center? Yes 5.13% N o 94,877 . 



Since my child has been in this special program I feel that he 

3, (a) enjoys school more than he used to, 69,177, 

(b) feels the same as he always did about school 28,33 

(c) likes school less than he did before 2,50 

4, (a) is more willing to go to school in the morning 52,897, 

(b) acts the same as ever about going to school in the morning.,. 44.63 

(c) is not as willing as before to go to school in the morning... 2,48 

5, (a) gets better grades than he used to 67,507, 



(b) gets about the same grades as always 31,67 

(c) gets worse graded than he used to 83 

6, (a) talks more about school than before 57.14% 

(b) talks about school about as much as he ever did 38.66 

(c) talks leso about school than he used to 4,20 

7, (a) brings more school work home than he did before 30,25% 

(b) brings echoolwork home about as often as before 60.50 

(c) brings less schcolwork home than he used to 9,24 

8, (a) talks more often about what he would like to do when he 

graduates from high school 52,10% 

(b) talks the same as ever about what he would like to do when 

he graduates from high school ,....*«, 45.38 

(c) says less about what he would like to do when he graduates 

from high school 2,52 
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TABLE 21: (continued) 



ITEM 



Percentage of 
Responses 



9. (a) is absent from school less than he used to be 54.557. 

(b) is absent from school about as often as before 43.6-4 

(c) Is absent from school more than before... 1.82 



10. (a) likes his special classes better than his regular classes... 68.387. 

(b) likes both kinds of classes the same.. 26.50 

(c) likes his regular classes better than his special classes... 5.13 



In my opinion, the Rapid Learning Center 



11. (a) should become a regular part of the school system. 

(b) doesn't make any difference in the school system... 

(c) should not be used as a part of the school program 

12. (a) is worth spending money for 

(b) is worth as much as any other school program 

(c) is a poor way to use the taxpayers 1 money. 



91.537. 

6.78 

1.69 

68.077. 

31.93 

.00 



13. (a) will help my child graduate from high school. 91.607. 

(b) will have no effect on my child's graduating from high school 7.56 

(c) could keep my child from graduating from high school ....... • .84 



14. (a) gives my child a better chance for getting ahead in life.... 93.287. 

(b) will have no effect on my child's future 5.88 

(c) may hinder my child's chances for getting ahead............. .84 



The information from the above table reveals that nearly all the 
parents are aware of the Rapid Learning Center program, but very few have 
visited the center. The parents, as a whole, are very much in favor of 
the program and dealre it to become a part of the regular school program. 
Several of the questions reveal responses that the parents feel the pro- 
gram has had some impact on their childrens' interests and performance in 
school, but there are still a large percentage of responses indicating 
that their children are not doing much differently than what they have 
been doing In the past. 
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Results on Permanency of Student Gains ; 
ment test was to be given In October, 1970* 
Phase I have been Invalidated, the permanent 
achievement tests will need to be considered 
of the dropout prevention program. 



The permanency achieve- 
Since the test results of 
gain made by students on 
in the Phase II operation 
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V, RECOMMENDATIONS 

1* Because frequent exit testing during Phase I has caused 

a number of problems, It Is recommended that exit tests be 
given only at the end of 80 hours and 160 hours of Instruction. 
The testing period should be correlated with the end of the 
semesters. Furthermore, It Is recommended that the 160-hour 
test be correlated with the regular spring testing program of 
the school district If at all possible. This will allow a 
continuity with the regular school program. 

2. During Phase I no specific guidelines were established 
concerning when a student would be exited from the rapid 
learning center program. It Is recommended that students In 
the learning center programs In Phase II be exited Into 
turnkey classes at the time they take the exit tests If they 
are within one-half grade level of their proper grade level. 

3. A number of problems arose because of lack of clarifi- 
cation of terms or guideline procedures. It Is recommended 
that the following definitions and procedures apply for Phase II 
operations! 

a. A student will be considered a dropout from the program 
If he or she leaves school or the program and does not 
reenter. Exceptions to this definition are: (1) if a 
student graduates from high school, (2) If a student Is 
drafted Into military service, (3) If a student Is physi- 
cally or mentally incapacitated (pregnancy excepted) to 
such an extent that he or she Is not able to participate 



O 
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In the project and attend school as certified by a 
licensed physician, (4) other reasons mutually agreed upon 
in vrlting by the project director and the contractor. 

b. The starting time for each student will be the first day 
the student enters the program* Any exception to this 
must be agreed upon by the project director and the con- 
tractor, and any such agreement must be made in writing. 

c. The target population for Phase II are students fulfilling 

the following entry criteria: (1) students in 1969-70 

RLC program who did not gain one or more grade levels in 
reading comprehension or mathematics, (2) seventh-grade 
students who are two or more grade levels deficient In 
reading or mathematics and who have an IQ of 75 or higher 
on an Intelligence test, and (3) all students In grades 
8-12 who are two or more grade levels deficient In reading 
or mathematics and who have an IQ score of 75 or higher 
and who did not participate in the 1969-1970 RLC program. 
The contractor must be willing to accept the conditions 
and process by which the taiget population are selected. 

If any question exists, It must be reconciled within 
fifteen days of the student's enrollment in the project 
according to a negotiation procedure agreed upon by the 
project director and the contractor. 

d. The ending time for the Instructional program for each 
student shall be the date when the student took the per- 
formance or exit test. If the strident takes more than 
one exit test, the latter test date shall be considered the 
ending date. 
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e. Actual Inst r uctional time is the time between the 

starting time and the ending time minus the amount of 
time that the student was absent from the instructional 
program. 

4. During the Phase I operation difficulties arose because 
the contractor submitted names of students to take exit tests 
late and then he often changed the list Just prior to the 
testing date. It is recocmended that the project director 
establish exit test dates and the contractor shall submit the 
names of students to take teets to the project evaluator 

ten (10) days prior to the date of the test. This shall be 
considered the official test list. 

5. It is recommended that the contractor not be allowed to 
use as a practice test any form of any test that is being 
used as a post test for payment purposes. 

6. It is recommended that the contractor submit monthly 
reports to the project director. The contents of the reports 
shall be mutually agreed upon by contractor and project 
director but must Include essential financial and evaluation 
data required In the request for proposal. 

7. During Phase I the information available on students in 
the program was limited. It is recommended that a more compre- 
hensive information system be developed to provide accurate 
data on each student in the program. The evaluation design 
night be used as the basis for determining what data need 

to be included in the system. 



O 
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8. Lack of personnel handicapped the Phase I operation. It 
Is recommended that additional personnel ba employed to help 
in giving tests, collecting information, working with turnkey 
teachers, working with parents, counseling with students, 
showing program to visitors, preparing dissemination infor- 
mation, writing reports, and generally to help the project 
director. 

9. To provide necessary and timely test results, it is 
recommended thot participating school districts in the program 
administer the planned standardized tests (achievement and 
intelligence) on a systematic time schedule. This schedule 
should establish testing dates that would provide test results 
when they are needed for identifying thi target population, 
choosing the sample to start in the learning center or turn- 
key programs, and provide evaluation information. 

10. It is recommended that contracts be made with the Internal 
evaluator, external auditor, and management support. The con- 
tract shall outline the specific responsibilities and roles 

of each contractor. 

11. To assist in the strengthening of the management procedures, 
it is recommended that the project director or his designate 

do the following: 

a. Establish an adequate record system concerning the operation 
of the dropout prevention program. The record system should 
include agendas and minutes of meetings, copies of all 
reports received and given, as well as Information about 
equipment, facilities, personnel, and problems encountered. 

O 
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b. Delegate and review with all personnel their respective 
functions snd responsibilities. This review should 
include information concerning the organizational struc- 
ture, channels of communications, required reports, and 
lines of authority. 

c. Hold regularly scheduled meetings with the staff, the 
advisory committee, and any other involved groups for the 
purpose of continuous plsnning, evaluating, and improving 
the operational process. 

d. Submit monthly reports to the school superintendents In 
the participating school districts 10 as to keep them 
involved and informed of the status of the program. 

e. Hold regularly scheduled meetings with the learning center 
and turnkey teachers to obtain process information and 
make needed adjustments in the program as problems occur. 

f. Plan and schedule parents' activities to Involve them in 
the educational process of their children. 

g. Prepare and disseminate public information data to 
interested groups, both within the participating school 
districts and cut9lde the project area. 

h. Establish a time and day when visitors can be accepted 
rather than allow them to continuously disrupt the program. 

12. To avoid "teaching for the teat”, it is recommended that 

the following be written in the contract with the contractors 

a. The contractor shall not Include in any of his instruc- 
tional materials any exercises that are the same as the 
test items used in the tests that will be used to determine 
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how much the contractor will be paid. The definition of 
"same" would be determined by the rules outlined on 
pages 58-59 of this report. Variations In these rules 
must be mutually agreed upon by contractor and fiscal 
agent in writing prior to the starting of this project. 

b. The intornal evaluator shall periodically make a quality 
control check of the instructional materials In use in 
the program to determine whether the contractor has ful- 
filled the requirements listed under "a". The quality 
control check shall bo done once a month and be based on 

a systematic random sample of the Instructional materials. 

c. If any single test Item, as defined in the rules, Is 
found in the quality control check, the contractor shall 
be penalized $1,000 for each exposed Item. 

13. Since the achievement test results for Phase I have been 

invalidated through teaching to the test, it ie recommended 
that the following alternatives be explored for possible 
settlement with the contractor: 

a. A legal review of the contract might be made to determine 
the responsibility of the fiscal agent to make payment 
under existing circumstances. 

b. Establish an assumption that contractor shall be paid on 
the same basis as what It costs the school district to 
provide Instruction. Settlement might then be made un an 
hourly rate basis or on a per student basis for the 
instructional services provided by the contractor. 
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c. Several possibilities have been considered to determine 
ways of making corrections for the effects of teaching 
for the test. Within the time and cost constraints 
Imposed upon the internal evaluator for making this 
evaluation report, the conclusions of this report are 
that test results are not valid due to the exposure of 
test items in the instructional material; and thus, a valid 
correction factor cannot be established. If the school 
board feels that time and costs are justified to further 
verify this fact, it is recommended that Step II of the 
recommendations made by the consultants of Educational 
Testing Service be explored. If from Step II a sufficient 
number of test items still remain unexposed, it is recom- 
mended that Step III of the consultants' recommendations 
be explored as the possible basis of payment. The inter- 
nal evaluators feel that Dr. Dyer's and Mr. Angoff's 
recommendations are statistically sourd. 

To provide the above information it is anticipated 
that it will take at least 30 days of time involving 
clerical work, statistical work, and perhaps computer 
time. Even if the analysis is made there is a high 
degree of probability that not enough unexposed items 
will be found to apply a correction factor and the cor- 
rection factor might very well decrease the recorded 
student gains. Therefore, it is recommended that if 
alternative "c" is selected as the basis of settlement 
the contractor be required to pay the costs. 
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14* It is reconmended that payment to the contractor for 

Phase II be based on (a) achievement gains made by students, 

(b) extent contractor attains performance objectives, and 

(c) the reductions of dropouts. The achievement gains should 
be measured by standardized tests; the performance objectives 
should be measured by developing an instrument using a pool 
of items submitted by the contractor; and the number of drop- 
outs can be assessed from the school records. 
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AVAILABLE AT TIME FUMED 



Memorandum fori DR. ANDREW 



Subjects Analyoie of Data to Determine the Datei July 8, 1970 

Effect of the Doroett Instructional 
Program on Educational Qrovth of 

Studonte in Texarkana Promt Henry S. Dyer 

William H. Angoff 



The purpose of this memorandum is to outline a procedure for evaluating 
the real gains of students who participated in the Toxarkena Project after 
Adjusting for the fact that certain items on the exit tests have been 
compromised by having been ueed essentially unchanged in tho instructional 
materials. 

1. Bie first step in the procedure is to nako some judgments about tbs 
"sameness'* of items that appeared in both the instructional materials and in 
tho exit tests. These we ch all call the "exposed" items. 

2. The second step is to verify the foregoing Jud entente by making a 
atatietical analysis to determine how the items identified as "exposed" 
behave in comparison with items not eo identified. If it is found that the 
exposed items had become easier, then the exit answer shoot for those students 
will bs rescored using only non- expo cod i testa to dote mine an increment 
attributable to growth equivalent to the increment that would have been 
obtained from the full-length test had no itoms become exposed. 

3. The third step ie to establish the magnitude of the real gains earned 
•by the students on the non-exposed items. 

Step I - Identification of Exposed Items 

This step involves a comparison of the instructional exercises (i.e., 
the programs, the progress checks, and the laminated panels with the items on 
the exit toate (ITBS end SRA) to identify thoee items judgod to bo the "sano" 
as the instructional exercises. The rules for judging "sameness" are as follow i, 

i * 
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A test Item and an instructional axerolse are to be considered the 
sane if 

(1) Their vordlng Is identical in all respects doapite a change in format* 

Example t A. Which of these is a way to find the circumference 
in inches of o oirole with a 6-inch diameter? 

(1) 3x3. lit (3) 3x3x3. It* 

(2) 6x3. lit (U) 2x6x3* lit 

B. Which of these is a way to find the ciroumforence 
in inohos of a circlo with a 6-inch diameter? 

(1) 3x3. lit (2) 6x3. lit (3) 3x3x3. lit 

(it) 2x6x3.11t 

(Ho to the change in arrangement of options) 

(2) The wording of the stern and the wording of the corroct option are 

identical) despite the fact that other options havo been changed. 

Examples A. Which of these is a way to find the circumference 
in incites of a circle with a 6-inch diameter? 

(1) 3x3. lit (3) 3x3x3. Ut 

(2) 6x3. lit (It) 2x6x3 .lit 

B. Which of these is a way to find the circumference 
in inches of a circlo with a 6-inch diameter? 

(1) 3*Htx3 (3) 3x2. lit 

(2) 6sr3. Ut (U) 2x6x2. Utl6 
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(3) The correct option is identical andthe tm in sense of the stem has 

boen retained despite a minor change in wording. 

Examples A. Vftiioh of these is a way to find the oircwiforence 
in inches of a circle with a 6-inch diameter? 

(1) 3x3.11* (3) 3x3x3.11* 

(2) 6x3. It* (1*) 2x6x3.11* 

B. The number of inches in the circuraforonce of a dr ole 
with a diameter of 6 inches io 

(1) 6x3.1 1* (3) 3x3x3. ll* 

(2) 3x3.11* (U) 6x6x3.11* 

(U) The main sense of the whole item has been rotained dospite the 

fact that it has been restated in the negative. 

Example* A. Which of thoso is a way to find the droumforenoe 
in inches of a circle with a 6-inch diemotar? 

(1) 3x3.11* (3) 3x3x3.11* 

(2) 6x3.11* (1*) 2x6x3. li* 

B. The number of inches in the circumference of a circle 
with a diameter of 6 inches io not 

(1) 6x3.11*16 (3) 3x3x3.11* 

(2) 6x? 2* (1*) 2x3x22. 

(5) The main sense of the stem has boen retained despite a minor change 

in wording; the correot option io identlo&l; but one or more incorrect 
options have been changed or omittod. 

Examples A. Which of these in a way to find the olrcumference 
in inches of a circle with a 6-inch diemetor? 

(1) 3x3.11* (3) 3x3x3.11* 

(2) 6x3.U* (1*) 2x6x3. 1U 
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. » Die number of inches in the circumference of a 
circle haring a 6-inch diameter can be found by 
vhlch one of these? 



(1) 3*22 ( 2 ) 6x3. 1U (3) 3 x3jO.Hi 

(6) The item has been changed from a multiple-choice to a truo-false 

item by retaining the stem of the multiple-choice item and incorporating 

in tho eton ons of the options (correct or inoorroct). 

Examples A. Which of the no is a way to find the circumference 
in inches of a olvclo with a 6 -inch diameter? 

(1) 3x3. ll* (3) 3x3x3. Ill 

(2) 6x3. lU (U) 2x6x3. 1U 



B. The number of inches in the circumference of a 
circle with a 6~inch diameter is 3x3x3. Hi 



o 



False 



^^TJruo 

C. A circle with a 6-inch diameter has a oircumference 
of 6x3 .lU inches. 

True False 

Since the search for exposed items (i.e., item3 that according to the 
rules are the "same" in both th*. instructional materials and the exit tests) 
would be a stupendous task if all tho instructional materials had to be checked 
against all the items in all forms of the exit tests, If becomes important to . 
determine a cut-off point beyond which no further search is required for the 
decision whether a given form of a given exit test must be rejected as invalid. 
This cut-off point can be determined by reference to the reliability of the 
exit test as reported in the publisher's manual. 



O 
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In order to make this determination! we have assumed that if the 
reliability of scores based on the residue of unexposed items of a form or 
an exit test drops below .60! then the test cannot be used at all to estimate 
the amount of progress a student has made. On the basis of this assumption! 
ve suggest that the following table be used to deoide when it becomes no 
longer necessary to search the instructional materials for exposed items in 
order to deolare a test form invalid: 



Stop Search Table 



If the publisher’s reported 
reliability of the test is: 


Then stop the search for 
exposed items when the 
' per cent of exposed total 
items on the test form drops 
below: 


.95 


7956 


.91* 


7h* 


.93 


70 % 


.92 


65* 


.91 


‘60* 


.90 


56* 


.89 


51* 


.88 • 


16 % 


.87 


hO* 


.66 


35* 


.85 


29* 



This table is based on the formula: 

r 8 (l-rt) 

C * 

where c » the proportion of usable items 

r 0 ■ the reliability of scores based on a subset of usable items after 
exposed items have been eliminated 

r^ ■ the reported reliability of the total test 

It assumss the requirement that rg must be at least .60 to provide adequate 

measurement. (See page 9 below for the formula for r s .) 

d 
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Step IX - Empirical Verification of the Results of Step I 

From Stop I two groups of toot items will havo been identified on each 
exit test formt (l) those ITDS or SRA items judged to be "exposed" items 
and (2) those items judged "unopposed." 

Two groups of students should now be identified for each form of the 

test* (1) an "experimental" group — students who went through the Dorsott 

instructional program; and (2) a "control" group -- studonts who were equated 

* 

with the experiment als in the fall of 196?, but who did not go through the 
Dorsett instructional program. 

The item data for students in the "control" group will be obtained 
from those answer shoots that represent the firs t time the student has been 
exposed to a given form of a given exit test. Thus* 

(1) If a student took Form X at entry and Form Y at exit, use 
his answer sheet for Form Y only. 

(2) If he took Form X at entry and only Form X at exit, do not 
use his answer sheets at all in the analysis. 

(3) If ho took Form X at entry, Form X as a first exit test, 
and Form Y as the next exit test, use only his answer sheet 
from Form Y. 

(2i) If he took Fbra X at entry. Form Y as his first exit test, 
and Form Y again as a later exit test, use only his answer 

sheet from the first administration of Form Y. 

0 

(«>) If he took Fbrm X at entry, Form Y ao his first exit test, 
and Form Z as a later exit test, use his answer sheets from 
both Form Y and from Form Z. 



(Note that these rules apply only to the "control" group. For the 
"experimental" group, use all available answer sheets for each form of 
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e test which has been used as an exit test. ) 
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lhe basic type of analysis will consist of determining a proportion-pass 

value for each item on every form of the tests used as an exit test, 

calculated separately for the "experimental" group and the "control" group. 

Thus, for each item under consideration in, pay, Form Y of a test, two 

proportion-pass values will be calculated, one for each of the two defined 

groups. Proportion-pass (p-value) is defined as thenumber of students at an 

exit testing who answer the item correctly, divided by the total number of 

Npas3 

students in that group. Thus, p° — . After each p-value has been 

"tot 

calculated, it will bo transformed to "delta," which is the normal deviate 
corresponding to the p-value, expressed with a mean of 13 and standard 
deviation of b. Thus, a p-valua of .16 becomes a delta of 17.0; a p-value 
of ,8b becomes a delta 9.0; a p-value of .7 6 becomes a delta of 10.2; 
a p-value of .b6 becomes a delta of 13. Uj etc. 

Once the pairs of del tat have been calculated for the two groups, the 
items should be separated into two t ypes ? those classified according to 
Step I above as "exposed" and those classified as "unexposed." A bivariate 
graph should then bs prepared with the deltas for the control group along the 
x-axis and the deltas for the experimental group along the y-axls. Plot the 
"exposed" items on the bivariate papor, and identify the points with x's . 

Plot the "unexposed" items on the sams paper and identify the points with 
circles (0) to distinguish them from the "exposed" items. 

The points on this plot will fom one or two elliptical patterns extending 
from the lower left-hand section of the page to the upper right-hand section. 
These items that are relatively difficult for both groups will appear in the 
upper right eeotion of the plotj items that are relatively eusy for boti groups 
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will Appear in the lower left section of the plot) items that are relatively 
easy for the control group but relatively difficult for the experimental 
group will appear in the upper left section) and, finally, items that are 
relatively difficult for the control group but relatively easy for the 
experimental group will appear in the lowor right section. If the "exposed" 
items are no easier (relatively speaking) for the experimental group than 
for the control group, the x-points will be entirely intermixed with the 
o-points, and it will be concluded that the exposure did not operate to b ias 
the results for tho instructed group on the form of that exit test and therefore 
its scores can be taken at face value . 

If, however, the x-points tond to fall lowor and to the right of the 
ellipco described by the o-points, then it can be concluded in respect to 
that form of that exit test that the special instruction which made use of 
the exit test items did in faot have a specific differential effect on those 
items, thereby invalidating them for use in Judging whether the instructional 
program had a general positive educational effeot. 

Clearly, the items in the two ellipses will voiy likely not be entirely 

i 

distinct) there will be eome overlap, even if the "exposed" and ** .inexpos ed" 
itoraa are behaving differently. As a general rulo, it will be the sofor 
course of action tc define all items as suspoot except those that are clearly 
non-uuBpect. For this purpose, draw a line through the general centroid of 
all points with a slope equal to the ratio determined by tho standard 
deviation of deltas for the experimental group divided by the standard 
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deviatlon of the deltas for the control group. The equation for thie line let 
Y - RX ♦ B, 

where Y represents tho axis for the experimental group, 

X represents the axis for the control group, 

R * R,D *ExpA^* D *Cont A 
® " ^ExpA * ^Cont A 

Place a ruler parallel to that line and nova it toward the upper lef t of 
tho page until only o -points — no x-points -- appear above ths ruler. Take 
the items represented by those o-points as the non-suspect, or usable, items 
for the third step of the analysis .» 

Stop III p8termi.nat 3.o n of R ules for Asses si ng Score Pain a 

*•— — mb *— tmfmm mm »j — — m— tm m t> w>wci ■ - — warn tm » i i ■ — u 

For each form of the tests used at exit-, determine the proportion of 

suspect items and the complementary proportion (o) of usable items. Calculate 

the reliability of the subset of usable items by the following formula! 

cr t 

r a “ i'Vro^)r t ’ 

where o ia the proportion of the usable items, r^. is the reliability of the 
full-length test, as roportod in the test manual for the grado under 
consideration, and r a is the reliability of the given subset of usable items. 

For example, if tho reliability of the full tost lo .93 and it has been found 
in the preceding analysis that 2/3 of the items are suspeot, with 1/3 non-suspect, 
.the application of the formula given above is as follows* 

r 8 lToT +(733 -1 . 00)(T937 

. »31 .31 

r s " 1.00-.62 " .38 

r fl ■ .816 

■wAn x-itera in the upper left section end beyond the general swarm of o-items 
is one that ohould be re-exordned to see whether there is any question about 
the judgment that it has been exposed. Similarly a o-iten in the lower right 
section and beyond the general swarm of x-items should be ro-examined for tho 
O oility that it was missed in the search for exposed items. 

ERIC 



If r a la bslow .80 for eny cutset of items used in an exit test, then 
there is no way to get sn adequate measure of student performance from that 
subsot. 

If r is .80 or above for any subset of items, then continue as follovs: 

(1) Assemble the papers for the co ntrol gr oup on that form of the test 
and ro score those papers for' th9 subset of items. Compute the mean and 
standard deviation on that subset (s) and on the total test (t). 

(2) Determine tho constants of the following equation, Which allows the 
conversion of scores on the subnet to eet-inAted scoroa on the full length test. 

i 

’ V **•♦ *» 

where $£ ■ estimated raw score on full length test, 

T g • observed score on subset of non-ospoaed items, 

P ■ r ty*oi and Q ■ Mt - Ptf a . 

Ht • Kean on full-length teat, 

• 8.D. on full-longth teot, 

M a » Moan on subset, 

fg • S.D. on subset. 

X4 example, the equation le 

* 

t t • 3.2T fl ♦ $. 1 , 

•nd if, for, trample, a student's score on the subset is 1 $ (T a ■ 1 $), then 
Tt ■ (3*2)(1J>) ♦ 5'.1 * ltd ♦ $.1 • *>3il. V&en rounded to thn nearest whole 
nusber, this score is Ibis score is en eatfiaato et what the student's 

total rau score would ha?e been if nono of the item in tho txll teat bed 

been expoeod. 

. % 

I * 

POOR ORIGINAL COPY • BIST 
AVAILABLE AT TIME fILMEO * 



POOR ORIGINAL COPY - BEST 
AVAILABLE AT TIME RlM r ' ^ 

(it should be noted that the foregoing procedure is based on the 
assumption that tho unespo3ed items are at leaot reasonably representative 
of all the itens on the exit test fom with respect to content, difficulty 
lovel, and level of discrimination. To the degree that this assumption does 
not hold the proceduro ie open to some question. It would be dosirable to 
check this assumption to the extont possible by an examination of the test 
content and hy an examination of the distribution of item difficulties of 
the subset of unopposed itens in comparison with the remaining exposed 
items, as observed in tho control group. ' Ihis check should be made if there 
io tlno.) 

Using the student's estimated raw score on the exit tsst, enter the 
table of grade equivalencies to detemino the estimated grade e<*iivalent 
score on the exit teat fom. Subtract his entry grade equivalent score from 
his estimated exit grade equivalent score to get his estimated grads 
equivalent gain (0 09 *). 

Inasmuch as 0 a8 t io derived tr n a less reliable measure of exit performance 
than could havo been poeaiblo had none of the items been exposed, some 
adjustment for this unreliability ehould be made in the w The need for 
this adjustment makes itself evident In the faot that the covrelation of grade 
level with tost porfonunc© is lower, and that the slope of the regression line 
of test performance on grade level is smaller, when the test is loss roll able. 

The result is that a given soore difference represents a greater grado equivalent 
difference than it should. Karoo ver, because of the lower correlation of 
grodo and test score (due to the lover tost reliability), there is greater 
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regression toward the mean. Therefore it would be expected that with a 
low-scoring group like the one under study he*e, higher OB estimates would 
bo made for its students than would be n?de if the test were more reliable, 
and thereby correlating higher with grade lovel. Howevor, the adjustment 
oallod for here appears to be a highly complex one, and therofore, it will 
not bo attempted* Suffice it to sey that the omission of the edjustswnt 
tends to overestimate the true grade equivalents for the students on the 
exit tests and makes it appear that they experienced sowsvhat groater 
educational growth than was actually the oaso. 



POOR ORIGINAL COPY * BEST 
AVAILABLE A1 TIME FILLED 




O 

ERIC. 



